05. Deciding on Metrics - Part I

Deciding on Metrics

From our user funnel, we should consider two things: where and how we
should split users into experiment groups, and what metrics we will use to
track the success or failure of the experimental manipulation. The choice of
unit of diversion (the point at which we divide observations into groups) may
affect what metrics we can use, and whether the metrics we record should be
considered invariant or evaluation metrics. To start, decide on a unit of
diversion and brainstorm some ideas for metrics to capture.

To be clear, the overall plan is to test the effect of the new homepage using a
true experiment; in particular, we'll be using an A/B testing framework. This
means that prospective users should be split into two groups. The control, or
'A' group, will see the old homepage, while the experimental, or 'B' group,
will see the new homepage that emphasizes the 7-day trial.

Unit of Diversion

By which means should we divide visitors into our experimental and control groups?
SOLUTION: Cookie-based diversion

Brainstorm Potential Metrics

QUESTION:

Write down some potential metrics or ways of objectively measuring things related to evaluating the success of the experiment. You don't need to decide on invariant or evaluation metrics here: that'll be discussed on the next page.

SOLUTION:

These answers need to be solved by yourself, I believe you can do it

Next Concept